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(54) Driver's arousal level estimating apparatus for vehicle and method of estimating arousal 
level 



(57) A driver's arousal level estimating apparatus for 
a vehicle includes a FFT signal processing section for 
obtaining respective frequency component powers by 
applying a frequency conversion to displacement quan- 
tities successively detected of the vehicle, an evaluation 
value calculating section for calculating an evaluation 



value from a ratio of an average value of the respective 
calculated frequency powers and a maximum value of 
frequency component powers in a specified frequency 
domain containing a predetermined lateral fluctuation 
frequency characteristic of a lowered arousal level and 
a judging section for judging a driver's arousal level 
based on said evaluation value. 
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Description 

[0001] The present invention relates to an apparatus 
and method of estimating an arousal level of a vehicle 
driver by particularly monitoring lateral displacements of 
a vehicle. 

[0002] A technology for preventing car accidents 
caused by the decline of driver's arousal level is one of 
most important developmental subjects of today from 
the view point of vehicle safety. In recent years, studies 
for detecting drowsiness or arousal level of vehicle driv- 
ers, developments of techniques for warning them 
against falling into drowsy state are vigorously under- 
taken. When driver's arousal level descends, especially 
when a vehicle travels at high speeds, serious accidents 
may happen. Even if he or she does not fall into such 
drowsy state, drowsiness induces him or her to an ab- 
sent-mi nded state which hinders a quick action for avert- 
ing accidents, this also leading to accidents. 
[0003] JP-A-5-58192 discloses a technique in which 
so-called "dozing at the wheel" is detected based on low 
frequency components in displacement quantity of a ve- 
hicle. In this disclosure, displacement quantities such as 
steering angle, lateral displacement of a vehicle and the 
like are monitored consecutively and low frequency 
components in frequency spectrum of the displacement 
quantity are extracted. On the other hand, low frequency 
components in frequency spectrum of displacement 
quantity at normal operations are memorized as sample 
data beforehand. Then, the low frequency components 
after a specified time elapses from the start of the vehi- 
cle are compared with those sample data. If a difference 
between the low frequency components and the sample 
data exceeds a predetermined value, it is judged that 
the driver dozes at the wheel. 

[0004] However, the sample data which are used for 
the judgment reference are ones collected under a cer- 
tain traveling condition (weather, road surface condi- 
tions, time, degree of traffic jams, vehicle speed and oth- 
ers). If the traveling condition changes, the judgment 
whether or not the driver is in a dozing state is accom- 
panied by errors. That is, this background art has a prob- 
lem that, when the traveling condition changes largely, 
it is difficult to obtain a correct judgment as to whether 
or not the driver is in a dozing state. 
[0005] The applicant of the present invention has al- 
ready proposed a method of estimating arousal level to 
solve such a problem in JP-A-2000-1 85575. According 
to the technique disclosed in this publication, quantities 
of displacement in a widthwise direction of the vehicle 
are detected consecutively and respective frequency 
component powers are obtained by applying frequency 
conversion to these quantities of displacement. Next, 
the frequency domain is divided into a low frequency do- 
main and a high frequency domain and an integration 
A1 of the frequency components powers in the low fre- 
quency domain and an integration A2 of the frequency 
components powers in the high frequency domain are 



obtained respectively. Then, evaluation values are cal- 
culated using these integrations A1 , A2 and an arousal 
level of a vehicle driver is judged based on the evalua- 
tion values. 

5 [0006] It is an object of the present invention to pro- 
vide an arousal level estimating apparatus capable of 
making an accurate judgment of drivers arousai level. 
[0007] To achieve the object, an arousal level estimat- 
ing apparatus comprises a power calculating means for 

10 calculating a frequency component power by applying 
a frequency conversion method to a lateral displace- 
ment quantity detected successively, a first evaluation 
value calculating means for calculating a first evaluation 
value based on a sum of the frequency component pow- 

15 er calculated by the power calculating means, a second 
evaluation value calculating means for calculating a 
maximum value of frequency component power in a 
specified frequency domain including a fluctuation fre- 
quency as a second evaluation value, an evaluation val- 

20 ue calculating means for calculating an evaluation value 
from a ratio of the first evaluation value and the second 
evaluation value and a judging means for judging a driv- 
er's arousal level based on the evaluation value. 

25 Fig. 1 is a block diagram showing an arousal level 
estimating apparatus according to the present in- 
vention; 

Fig. 2 is a flowchart of a routine for calculating eval- 
uation values; 

30 Fig. 3 is a diagram showing a time-versus change 
of lateral displacement; 

Fig. 4 is a graph showing respective powers of fre- 
quency components; 

Fig. 5 is an explanatory view showing calculations 
35 of evaluation values; 

Fig. 6 is a flowchart of a routine forjudging warning; 
and 

Fig. 7 is a diagram showing actual measuring re- 
sults in traveling on a high way. 

40 

[0008] Referring now to Fig. 1, reference numeral 1 
denotes a lateral displacement detecting section for de- 
tecting a displacement in a widthwise direction of the 
vehicle (hereinafter referred to as lateral displacement). 

45 For the purpose of the lateral displacement detecting 
section 1 , for example, a stereoscopic camera using 
CCD and the like or a monocular camera can be used. 
An image information processing section 2 serves as 
processing images acquired by the lateral displacement 

50 detecting section 1 and obtaining quantities of displace- 
ment of the vehicle. For example, left and right lane 
markers of roads are imaged by a CCD camera and im- 
age data of one frame are stored in a memory of the 
image information processing section 2. Then, the left 

55 and right lane markers are recognized respectively by 
utilizing image recognition technique. That is, an area 
corresponding to the left and right lane markers is iden- 
tified from the image data of one frame using known 



3NSDOCID: <EP 1 20901 9A2_I_> 



3 

methods of recognition like a template method or ster- 
eomatching method. The position of the own vehicle 
within the left and right lane markers can be calculated 
from the lateral distance from a center of the vehicle to 
a center of the left and right lane markers. s 
[0009] Besides the CCD camera, lateral displace- 
ment can be detected also by a vehicle-road communi- 
cation system using magnetic nails buried in roads, a 
GPS navigation system combined with vehicle speeds 
and the like (regarding a technique of warning using a 10 
navigation system, see Japanese Patent Application 
Laid-open No. Toku-Kai-Hei 9-99756). Also, lateral dis- 
placement can be estimated by steering angles and the 
lateral displacement detecting section 1 can be replaced 
with a steering angle sensor. Furthermore, lateral dis- 15 
placement may be estimated by detecting yaw rate or 
lateral acceleration. Fluctuations or displacement quan- 
tities in the lateral direction of the vehicle are measured 
with resolution of 1 .0 millimeter at a time interval of 0.1 
seconds. Displacement related data is stored in a shift 20 
register of a FFT signal processing section 3 for obtain- 
ing frequency components power. A group of displace- 
ment data calculated in time series and accumulated for 
a specified time are stored. These data are updated suc- 
cessively by newly calculated data. 25 
[0010] The FFT signal processing section 3, an eval- 
uation value calculating section 4 and a judging section 
5 are constituted by CPU, RAM, ROM and input and out- 
put circuits respectively and act as functional units in the 
computer. These functional units 3 to 5 are realized 30 
when respective components of the computer operate 
on each other under the control of an application pro- 
gram for executing a routine which will be described 
hereinafter. In the ROM, programs, a lower limit value 
PLOW, a table for establishing a step value a, warning 35 
judging values D1 , D2 and the like are stored. 
[0011] Fig. 2 is a flowchart of a routine for calculating 
evaluation values. This routine is executed repeatedly 
ata specified time interval. At a step 1 , the displacement 
data acquired for the past X seconds and stored in the *o 
shift register of the FFT signal processing section 3 are 
read out every Y seconds (for example, 90 seconds). 
The sampling time X should be established at a rather 
long time (for example 50 to 80 seconds) in order to 
make an accurate estimation of arousal level. 
[0012] At a step 2, the FFT signal processing section 
3 applies a frequency conversion to the displacement 
quantity using a fast Fourier transformation (FFT) and 
the like and calculates respective frequency compo- 
nents power (amplitude) P[i] in a frequency spectrum, so 
In case of this embodiment, 16 frequency components 
powers P[1 ] to P[1 6] are calculated at an interval of 0.02 
[Hz] in a frequency domain of 0.03 to 0.3[Hz]. The rea- 
son why the frequency domain below 0.03Hz is not tak- 
en into consideration is that powers of that domain tend 55 
to increase when the vehicle travels on a curved road 
and the frequency domain has no direct relationship with 
driver's arousal level. Further, the reason why the fre- 



4 

quency domain above 0.3Hz is not taken into consider- 
ation is that since normally powers are negligibly small 
at that frequency domain, the amount of calculation of 
evaluation value H can be reduced. 
[0013] The relationship between displacement quan- 
tity and frequency component power will be described. 
Fig. 3 is graphs showing the relationship between the 
change of lateral displacement quantity and elapsed 
time from the start of a vehicle. These graphs show the 
result of measurement when the vehicle travels on a rel- 
atively less crowded high way under a relatively monot- 
onous traveling condition. The graph "After 10 minutes" 
shows a state immediately after the vehicle enters into 
the traffic stream and the displacement quantity is still 
small. When 20 minutes elapses, a driver gets accus- 
tomed to the traveling condition and becomes relaxed. 
As a result, compared with the graph "After 1 0 minutes", 
low frequency components increases and high frequen- 
cy components decreases. Further, when 50 minutes 
elapses, the driver is bored and becomes slightly dozy. 
As a result, occasionally, a large displacement tends to 
generate. Compared with the graph "After 20 minutes", 
there is an outstanding tendency that the displacement 
quantity of low frequency components increases. 
[0014] Fig. 4 is a graph showing a relationship be- 
tween a frequency component i and a power P[i] at the 
frequency component I, when the displacement quantity 
for each elapsed time of Fig. 3 is subjected to frequency 
conversion and a graph expressing discrete respective 
frequency component powers P[i] as broken line. In this 
graph, a dotted line shows frequency component pow- 
ers P[i] when 10 minutes elapse after the start of the 
vehicle, a broken line shows frequency component pow- 
ers P[i] when 20 minutes elapse after the start of the 
vehicle, and a solid line shows frequency powers P[i] 
when 50 minute elapse afterthe start of the vehicle. The 
graph indicates that frequency component powers P[i] 
of the low frequency domain has a tendency to increase 
as traveling time is elongated. 

[0015] At a step 3, respective frequency component 
powers P[i] (i= 1 to 1 6) in a frequency domain of 0.03 to 
0.3[Hz] are subjected to a leveling process according to 
the following formula to obtain leveled frequency com- 
ponent powers P'[i]. 

[Formula 1] 
P'[i] = P[i].f n 

where exponent n: 2.0 n 3.0; f = frequency. 
[0016] Assuming that the fluctuation of the vehicle 
within a lane is one of various fluctuations existing in the 
natural world, the amplitude of the fluctuation is 1/f and 
the power of the fluctuation is 1/f 2 . Accordingly, the ex- 
ponent n of the formula 1 is theoretically 2. However, 
according to experiments, it has been found that n = 2.5 
is most preferable. The reason why n = 2.5 is considered 



EP 1 209 019 A2 



50 



3 



BNSDOCID: <EP 1209019A2J_> 



5 



EP 1 209 019 A2 



6 



to be ascribed to vehicle and human factors, the effect 
of roads or the like. Nevertheless, it is possible to judge 
an arousal level using an arbitrary exponent n ranging 
from 2.0 to 3.0. This embodiment adopts 2.5 for expo- 
nent n. 

[001 7] Fig. 5 shows a relationship between frequency 
components i and leveled frequency component powers 
P'[i]. An overall feature can be visually recognized from 
a distribution of the leveled frequency powers P'[i]. The 
frequency component power after 50 minutes indicates 
that a power P'[4] at 0.09Hz and a power P'[5] at 0.1 1 Hz 
of a low frequency domain abruptly increase in vicinity 
of a fluctuation frequency f1 (= 0.1 Hz). When driver's 
arousal level is at a low level, there is a tendency that 
with respect to lateral displacement of the vehicle the 
powers near the fluctuation frequency f1 increase. In 
other words, when an arousal level is low, there is a fea- 
ture that only powers in a low frequency domain includ- 
ing the fluctuation frequency f1 increases and powers 
except that domain are low. In view of these tendencies, 
driver's arousal level can be judged by comparing a 
peak of powers near the fluctuation frequency f 1 with 
powers at frequency domains other than the fluctuation 
frequency f1 . 

[0018] The fluctuation frequency f1 is defined as a fre- 
quency which increases when driver's arousal level is 
low (including a dozing condition). Generally, in case of 
passenger cars, the fluctuation frequency tends to ap- 
pear at at a frequency domain around 0.08 to 0.12 Hz. 
In actual cases, since the fluctuation frequency is affect- 
ed by a response lag of vehicle behavior accompanied 
by steering operations, vehicle characteristics, vehicle 
speeds and the like, appropriate values are established 
for each model through experiments or simulations. In 
this embodiment, the fluctuation frequency f1 is estab- 
lished to 0.1Hz. 

[0019] At a step 4, the evaluation value calculating 
section 4 obtains a grand total of respective frequency 
component powers P'[1] to P'[1 6] and calculates an av- 
erage value P' ave . According to the embodiment, in or- 
der to correctly reflect driver 1 s arousal level on an eval- 
uation value H which will be described hereinafter, a 
maximum power being excluded from the frequency 
powers P'[1] to P'[16], the average value P' ave is calcu- 
lated from the rest of frequency powers P*[1] to P'[16]. 
The reason why thus filtering is performed is that the 
effect of increasing power of the fluctuation frequency 
f 1 and the effect of disturbances are deleted. 
[0020] At a step 5, the evaluation value calculating 
section 4 judges whether or not the average value P' ave 
is smaller than a lower limit value PLOW (for example 
100) established beforehand. In case where the aver- 
age value P' ave is smaller than the lower limit value 
PLOW, the arousal condition of the driver is judged to 
be stable and the lower limit value PLOW is set to the 
average value P' ave (step 6). This is for preventing the 
evaluation value H from growing unreasonably large 
when the evaluation value H is calculated at a step 10. 



On the other hand, in case where the average value 
P' ave is larger than the lower limit value PLOW, the pro- 
gram skips to a step 7. 

[0021] At the step 7, the evaluation value calculating 
5 section 4 makes a comparison of sizes between fre- 
quency component powers P'[4] and P'[5] in a specified 
frequency domain (0.09 to 0.1 1 Hz). That frequency do- 
main includes a fluctuation frequency f 1 (0.01 Hz). Then , 
a larger one is set as a fluctuation frequency power P' slp . 
10 That is, in case where the power P'[5] at 0.11 Hz is larger 
than the power P'[4] at 0.09Hz, the power P'[5] is set to 
the fluctuation frequency power P' s)p (step 8). On the 
other hand, in case where the power P'[5] at 0.11 Hz is 
smaller than the power P'[4] at 0.09Hz, the power P'[4] 
15 is set to the fluctuation frequency power P' slp (step 9). 
[0022] At a step 10, the evaluation value calculating 
section 4 calculates an evaluation value H according to 
the following formula: 

20 

[Formula 2] 
H=PV P 'aveXl00 

25 This evaluation value is considered to be a momentary 
arousal level not including a time factor and is calculated 
as a ratio of the maximum power P' S | p at the frequency 
domain 0.09 to 0. 1 1 Hz to the average value P' ave of re- 
spective frequency component powers P'[i]. After the 

so evaluation value H is calculated at the step 1 0, the pro- 
gram leaves the routine. As shown in Fig. 5, when 10 
minutes elapse, since the power P' slp (P'[4] or P'[5]) is 
small; the evaluation value H is also small. On the other 
hand, when 50 minutes elapse, a driver's arousal level 

35 goes down. As a result, the power P' slp increases and 
the evaluation value H becomes large. Thus, it is under- 
stood that the evaluation value H has a strong correla- 
tion with the driver's arousal level. 
[0023] Fig. 6 is a flowchart of a routine for judging 

40 warnings. This routine is repeatedly executed at a spec- 
ified interval. First, at a step 21, the judging section 5 
establishes constants a1 to a8, 0 as step values accord- 
ing to the following table based on an evaluation value 
H calculated in the evaluation value calculating routine. 

45 These constants are for varying a value of an arousal 
level counter D and has a nonlinear relationship like 
Ia1l>la2l>lu3l>ltt4l>la5l,ltt6l<la7l<ltt8l. 

(Establishment of step values) 

50 

[0024] 



55 



Evaluation value 


Step value 


> 1000 


+a1 


>900 


+a2 


>800 


+a3 
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(continued) 



Fvatuation value* 


Step value 


> 500 


+a4 


> 400 


+a5 


> 300 


±0 


> 200 


-a6 


> 100 


-a7 


> 0 


-a8 



[0025] Next, at a step 22, the judging section 5 up- 
dates the value of the arousal level counter D by adding 
or subtracting a step value a to orfrom the present value 
of the arousal level counter D. Then, at a step 23, it is 
judged whether or not the updated value of the arousal 
level counter D is larger than a first judging value D1 (1 st 
warning judgment). If the updated value of the arousal 
level counter D is smaller than the first judging value D1 , 
it is judged that the driver is in an arousal state and the 
program leaves the routine. On the other hand, if the 
updated value of the arousal level counter D is larger 
than the first judging value D1 , the program goes to a 
step 24. 

[0026] At the step 24. it is judged whether or not the 
value of the arousal level counter D is larger than a sec- 
ond judging value D2 (2 nd warning judgment) . If the val- 
ue of the arousal level counter D is smaller than the sec- 
ond judging value D2, the program goes to a step 25 
where, in order to warn fluctuations of the vehicle, it is 
instructed to the warning section 6 that a first warning 
should be issued and leaves the routine. On the other 
hand, if the value of the arousal level counter D is larger 
than the second judging value D2, the program goes to 
a step 26, where in order to warn the driver in a dozing 
state, it is instructed to the warning section 6 that a sec- 
ond warning should be issued and leaves the routine. 
[0027] The earning section 6 executes miscellaneous 
warning processes to urge the driver to wake up re- 
sponding to the instruction from the judging section 5. 
There is a variety of warning processes. One example 
is to sound an alarm. In case where it is judged that the 
arousal level descends, an reference inter-vehicle dis- 
tance may be established to a rather long value. The 
warning section 6 may sounds a lane running-off alarm 
in a rather early timing or sounds a dozing alarm directly. 
Further, when the arousal level descends, the warning 
section 6 may instruct to display "Car fluctuates" on a 
dashboard of the vehicle. 

[0028] Fig. 7 is a graph showing an actual measure- 
ment result when a vehicle travels at high speeds. In 
which, a lower diagram Indicates a characteristic of lat- 
eral displacement of a vehicle, upper one indicates a 
characteristic of evaluation values H and a middle one 
indicates a characteristic of the arousal level counter D, 
respectively. According to this measurement result, 
when around 1400 seconds elapse from the start of the 
vehicle, peaks characteristic of lateral displacement ap- 



pear successively and the fluctuation frequency f1 (= 
0.1Hz) is found. As a result, the evaluation value H in- 
creases and the value of the deviation counter D is in- 
cremented. When this increment occurs successively, 

5 an alarm is raised to the driver at an appropriate timing. 
According to the characteristic of lateral displacement 
of Fig. 7, the peaks appear before 1400 seconds. How- 
ever, according to the embodiment, unless such peaks 
appear successively (unless the arousal level counter 

10 D is successively incremented), the warning section 6 
does not raise a warning. 

[0029] Thus, according to the embodiment, the driver' 
s arousal level is judged by comparing the peaks of pow- 
ers around the fluctuation frequency f 1 with the powers 
15 of other frequency domains. Therefore, the driver's 
arousal level can be judged only on the basis of the 
present and previous data. As a result, the arousal level 
can be properly judged even if the traveling condition 
changes. 

20 [0030] In calculating the evaluation value H, with re- 
spect to the average value P'ave of the frequency pow- 
ers P'[l], since the lower limit value is established in or- 
der to prevent the denominator of the formula 2 from 
becoming excessively small the arousal level can be 

25 estimated accurately without having adverse effects of 
small disturbances when the vehicle travels at high 
speeds and driving patterns inherent to individual driv- 
ers. 

[0031] Further, in this embodiment, when peaks of 
30 powers in the frequency domain including the fluctuation 
frequency f1 stand out against powers of other frequen- 
cy domains, the lowered arousal level is detected. This 
method provides an advantage of preventing erroneous 
judgments of the arousal level even when lateral dis- 
35 placement quantity is small, or even when the vehicle 
has a light side wind or even when a large vehicle pass- 
es by. 

[0032] Further, according to a prior art, a final arousal 
level is calculated from a time average of sporadic 

40 arousal levels and the judgment of warning is made by 
comparing this final arousal level is compared with a 
threshold value. However, this method according to the 
prior art has a problem that the warning is accompanied 
by time lag. On the other hand, according to the method 

45 of the present invention, in case where an evaluation 
value H corresponding to sporadic arousal levels is 
large (particularly in case where the arousal level is sub- 
stantially low), the step value a of the arousal level coun- 
ter D is established to a large value. Accordingly, the 

so warning can be issued without time lag. 

[0033] While the present invention has been dis- 
closed in terms of the preferred embodiment in order to 
facilitate better understanding of the invention, it should 
be appreciated that the invention can be embodied in 

55 various ways without departing from the principle of the 
invention. Therefore, the invention should be under- 
stood to include all possible embodiments which can be 
embodied without departing from the principle of the in- 
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vention set out in the appended claims. 



Claims 

5 

1. An arousal level estimating apparatus for estimat- 
ing an arousal level of a driver of a vehicle compris- 
ing: 

a power calculating means for calculating a fre- 10 
quency component power by applying a fre- 
quency conversion method to a lateral dis- 
placement quantity detected successively; 
a first evaluation value calculating means for 
calculating a first evaluation value based on a is 
sum of said frequency component power cal- 
culated by said power calculating means; 
a second evaluation value calculating means 
for calculating a maximum value of frequency 
component power in a specified frequency do- 20 
main including a fluctuation frequency as a sec- 
ond evaluation value; 

an evaluation value calculating means for cal- 
culating an evaluation value from a ratio of said 
first evaluation value and said second evalua- 25 
tion value; and 

a judging means for judging a driver's arousal 
level based on said evaluation value. 

2. The apparatus according to claim 1 , wherein 30 

said frequency component powers are sub- 
jected to a leveling process. 

3. The apparatus according to claim 2, wherein 

said leveled frequency component power is 35 
obtained by multiplying said frequency component 
power by a frequency raised to n power. 

4. The apparatus according to claim 3, wherein 

said exponent n is a value not more than 3 40 
and not less than 2. 

5. The apparatus according to claim 1, 2, 3 of 4, 
wherein 

said first evaluation value is obtained based «s 
on said frequency component powers excluding a 
maximum one from said respective frequency com- 
ponent powers calculated by said power calculating 
means. 

50 

6. The apparatus according to claim 1 , 2, 3, 4 or 5, 
wherein 

said evaluation calculating means calculates 
said evaluation value in time sequence and said 
judging means increases or decreases a counter 55 
value according to said evaluation value and when 
said counter value reaches a judging value, judges 
that a warning should be raised to a driver of said 



vehicle. 

7. The apparatus according to claim 6, wherein 

said judging means changes said counter val- 
ue according to said evaluation value. 

8. An arousal level estimating method, comprising the 
steps of: 

.calculating a frequency component power by 
applying a frequency conversion method to a 
lateral displacement quantity detected succes- 
sively; 

calculating a first evaluation value based on a 
sum of said frequency component power cal- 
culated by said power calculating means; 
calculating a maximum value of frequency 
component power in a specified frequency do- 
main including a fluctuation frequency as a sec- 
ond evaluation value; 

calculating an evaluation value from a ratio of 
said first evaluation value and said second 
evaluation value; and 

judging a driver's arousal level based on said 
evaluation value. 

9. The method according to claim 8, wherein 

said step of calculating said first evaluation 
value is obtaining said first evaluation value based 
on said frequency component powers excluding a 
maximum one from said respective frequency com- 
ponent powers calculated by said step of calculat- 
ing said frequency component power. 

10. The method according to claim 8 or 9, wherein 

said step of calculating said evaluation value 
is calculating said evaluation value in time se- 
quence and said step of judging is increasing or de- 
creasing a counter value according to said evalua- 
tion value and when said counter value reaches a 
judging value, is judging that a warning should be 
raised to a driver of said vehicle. 
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FIG.2 
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